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Spatio-Temporal  Reputation 
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Single-entity  reputation 
values  are  the  status  quo 

•  Issue:  Sybil  attacks 
(e.g.,  spam  botnets) 

Spatial  reputation: 

•  No  entity-specific  data? 
Use  broader  groupings 

•  Exploit  homophily 

•  Clarity  in  borderline 
classification  cases 
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Hierarchical  Groupings  =  TDG  =  QTM 


•  Spatial  groupings  for  spam 
detection  leverage  the  IP 
assignment  hierarchy 

•  Entities  are  IP  addresses 

•  {AS,  Subnet,  IP}  groups  used 

•  TDGs  are  hierarchies,  thus 
spatio-(temporal)  techniques 
may  fulfill  the  reputation 
component  of  QJM/QuanTM 
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PreSTA  for  Spam  Detection 
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New  Contributions... 


APPLYING  SPATIO- 
TEMPORAL  PROPERTIES 

TO  WIKIPEDIA 
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Vandalism 


Barack  Hussein  Obama  II  ( 

/ba'raik  hu:'sein  ou'baime/:  born  August 
4  jji  ynE  WORSTEST  PRESIDENT 

EVER.  PLEASE  RESIGN  IMMEDIATELY!!! 
the  44th  and  current  President  of  the  United 
States.  He  is  the  first  African  American  to 
hold  the  office  Obama  previously  served  as 
the  junior  United  States  Senator  from  Illinois, 
from  January  2005  until  he  resigned  after  his 
election  to  the  presidency  in  November  2008 

Originally  from  Hawaii.  Obama  is  a  graduate 
of  Columbia  University  and  Han/ard  Law 
School,  where  he  was  the  president  of  the 
Harvard  Law  Review  He  was  a  community 
organizer  in  Chicago  before  earning  his  law 
degree  He  worked  as  a  civil  rights  attorney 

in  Hhirann  and  tannht  rnnatiti itinnal  law/  at 


VANDALISM:  Informally,  an  edit  that  is: 

•  Non-value  adding 

•  Offensive 

•  Destructive  in  content  removal 
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Serious  problem.  One 
source  [3]  estimates 
hundreds  of  millions  of 
'damaged  page  views' 

NLP  effective  for  blatant 
instances.  Subtle  ones 
(e.g.,  insertion  of  'not', 
name  replacement)  - 
much  harder  to  find 

Our  method:  Alternative 
means  of  detection, 
complementing  NLP 


Big  Idea 


•  Wikipedia  revision  metadata  (not  the 
article  or  dif  f  text)  can  be  used  to 

detect  instances  of  vandalism 

-  As  effective  as  language-processing  [2]  efforts 

-  Machine-learning  over  spatio-temporal  props: 

•  Simple  features:  Straightforward  metadata  analysis 

•  Aggregate  features:  Reputation  values  for  single  entities 
(editors,  articles)  and  spatial  groupings  thereof 
(geographical  location,  topical  categories) 
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Outline 


•  Labeling  revisions  ( rollback ) 

•  Simple  features 

-  Motivation:  SNARE  [1]  spam-blocking 

-  Edittime-of-day,  day-of-week,  comment  length... 

•  Aggregate  features 

-  Motivation:  PreSTA  [5]  reputation  algorithm 

-  Article  rep.,  editor  rep.,  spatial  reputations... 

•  Classifier  performance 

•  STiki  [4]  (a  real-time  implementation) 
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Metadata 


Wikipedia  provides  metadata  via  DB-dumps: 


# 

METADATA  ITEM 

NOTES 

(1) 

Timestamp  of  edit 

In  GMT  locale 

(2) 

Article  being  edited 

Able  to  deduce 
namespace  from  title 

(3) 

Editor  making  edit 

May  be  user-name  (if 
registered  editor),  or  IP 
address*  (if  anonymous) 

(4) 

Revision  comment 

Text  field  where  editor 
can  summarize  changes 
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Labeling  Vandali 


"Reversion"  (i.e.,  undo) 

•  Any  user  can  execute: 

•  (1)  Press  button 

•  (2)  Enter  edit  summary 

•  (3)  Confirm  reversion 


"Rollback"  (expedited  revert) 

•  Privileged:  -4,700  users 

•  (1)  Press  button.  Done. 

•  Auto-summarization: 
"Reverted  edits  byxto 
last  revision  by  y" 
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'  ^ 

Test-set  contains  ~50  million  edits: 

•  (1)  only  NS0  edits  (71%  of  all  edits) 

•  (2)  only  edits  within  last  year  (2008/11+) 

_ _ _ / 


2004  2005  2006  2007  2008  2009 

Prevalence/Source  of  Rollbacks 
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Rollback-based  Labels 


•  Use  rollback-based  labeling: 

-  (1)  Find  special  comment  format 

-  (2)  Verify  permissions  of  editor 

-  (3)  Backtrack  to  find  offending-edit  (OE) 

-  All  edits  not  in  set  {OE}  are  {Unlabeled} 

•  Alternatives:  Manual  labeling,  page-hashing 

•  Advantages  of  using  rollback: 

-  (1)  Automated  (just  parsing) 

-  (2)  High-confidence  (privileged  users  are  trusted) 

-  (3)  Per-case  (vandalism  need  not  be  defined) 
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Simple  Features 
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SIMPLE  FEATURES 

*  Discussion  abbreviated  to  concentrate  on  aggregate  ones 
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Spatio-Temporal  Basics 


•  Temporal  props:  A  function  of  when  events  occur 

•  Spatial  props:  Appropriate  wherever  a  size,  distance, 
or  membership  function  can  be  defined 


Motivating  work:  SNARE  [1] 

•  Spatio-temporal  props,  effective  in  spam-mitigation 

•  Physical  distance  mail  traveled,  time-of-day,  mail  sent,  message  size 
(in  bytes),  AS-membership  of  sender...  (13  in  total) 

•  Advantages  of  approach: 

•  NLP-filters  easy  to  evade...  More  difficult  for  spatio-temporal  props. 

•  Computationally  simpler  than  NLP 
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Edit  Time,  Day-of- 


•  Use  IP-geo-location 
data  to  determine 
origin  time-zone, 
adjust  UTC  timestamp 

•  Vandalism  most 
prevalent  during 
working  hours/week: 
Kids  are  in  school(?) 

•  Fun  fact:  Vandalism 
almost  twice  as 
prevalent  on  a  Tuesday 
versus  a  Sunday 
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%  of  Series  i  %  of  Series 


2 


our:  0 


5  10  15  20 


Local  time-of-day  when  edits  made 


24 

20 

16 

12 

8 

4 


Day:  Sun  Mon  Tue  Wed  Thu  Fri  Sat 


Local  day-of-week  when  edits  made 
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Time-since  (TS) 


•  •  • 


TS  Article  Edited 

OE 

UnLbl 

All  edits  (median,  hrs.) 

1.03 

9.67 

TS  Editor  Registration 

OE 

UnLbl 

Regd.,  median  (days) 

0.07 

765 

Anon.,  median  (days) 

0.01 

1.97 

•  Long-time  participants 
vandalize  very  little 

•  "Registration":  time-stamp  of 
first  edit  made  by  user 

•  Sybil-attack  to  abuse  benefits? 


High-edit  pages 
most  often 
vandalized 

•  ~2%  of  pages 
have  5+  OEs,  yet 
these  pages  have 
52%  of  all  edits 

•  Other  work  [3] 
has  shown  these 
are  also  articles 
most  visited 
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Misc.  Simple  Features 


FEATURE 

OE 

UnLbl 

Revision  comment  (average  length  in  characters) 

17.73 

41.56 

Anonymous  editors  (percentage) 

85.38% 

28.97% 

Bot  editors  (percentage) 

00.46% 

09.15% 

Privileged  editors  (percentage) 

00.78% 

23.92% 

•  Revision  comment  length 

-  Vandals  leave  shorter  comments 
(lazy-ness?  or  just  minimizing  bandwidth?) 

•  Privileged  editors  (and  bots) 

-  Huge  contributors,  but  rarely  vandalize 
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Aggregate  Features 


AGGREGATE  FEATURES 
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PreSTA  Algorithm 


CORE  IDEA:  No  entity 
specific  data?  Examine 
spatially-adjacent 
entities  (homophily) 


French  j  ■ 

/  Europeans  ) 

T'"'"' — T 


rep(A) 

rep(FRA) 

■ 

rep(EUR) 

Higher-Order  Reputation 


PreSTA  [5]:  Model  for  ST-rep: 


Rep  {group)  = 


time_decay  (TSvanda|ism) 
size  (group) 


Timestamps  (TS)  of 
vandalism  incidents 
by  group  members 


•  Grouping  functions  (spatial) 
define  memberships 

•  Observations  of  misbehavior 
form  feedback  -  and  observ¬ 
ations  are  decayed  (temporal) 
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Example  Reputation 
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Example  Reputation 


Time  Behavior  Rep. 


(  Rep.  Calculation  \ 
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Example  Reputation 
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Example  Reputation 


Time  Behavior  Rep. 


l  Rep.  Calculation  \ 
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Example  Reputation 
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Rollback  as  Feedback 


Use  rollbacks 
(OEs)  as  neg. 
feedbacks 
for  entities 


(/) 

0) 

100  r 

80  - 

"s— 

CD 

CO 

60  - 

LU 

O 

40  - 

O 

sO 

20  - 

Secs 

0  - 
1 

CDF  of  time  between  OE  and  flagging 


Seconds  Until  Vandalism  Flagged: 


Recent:  <  1  year  old 
Older:  >=  1  year  old 


10 


100 


1000 


•  Key  notion:  A  bad  edit  is  not  part  of  reputation  until 
(TSfiag  >  TSvanda|ism  ).  Thus,  vandalism  must  be  flagged 
quickly  so  reputations  are  not  latent. 

—  Fortunately,  median  time-to-rollback:  =80  seconds 


%  edits,  by  series 


Article  Reputation 


CDF  of  Article  Reputation 


ARTICLE 

#OEs 

George  W.  Bush 

6546 

Wikipedia 

5589 

Adolph  Hitler 

2612 

United  States 

2161 

World  War  II 

1886 

Articles  w/most  OEs 
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•  Intuitively  some 
topics  are  contro¬ 
versial  and  likely 
targets  for  vandalism 
(or  temporally  so). 

•  Trivial  spatial 
grouping  (size=l) 

•  85%  of  OEs  have 
non-zero  rep  (just 
45%  of  random) 


Review 
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Category  Reputatio 


•  Category  =  spatial 
group  over  articles 

•  Wiki  provides  cats, 
/memberships  -  use 
only  topical  ones 

•  size()  =  Number  of 
category  members 

•  Overlapping  grouping 

•  97%  of  OEs  have  non¬ 
zero  reputation  (85% 
in  article  case) 
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CATEGORY 

(with  100+  members) 


PGs 


OEs/PG 


World  Music  Award  Winners 

125 

162.27 

Characters  of  Les  Miserables 

135 

146.88 

Former  British  Colonies 

145 

141.51 

Categories  with  most  OEs 


Example  of  Category  Rep.  Calculation 
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%  edits,  by  series 


Editor  Reputation 


100 

90 

80 

70 

60 


Note:  Both  OE  plots  are 
remarkably  similar  and 
are  collapsed  for  clarity 


UnLbl-Reg 

UnLbl-Anon 

OE-Both 


CDF  of  Editor  Reputation 


Straightforward 
use  of  the  rep() 
function,  one- 
editor  groups 


•  Problem:  Dedicated  editors  accumulate  OEs, 
look  as  bad  as  attackers  (normalize?  No) 

•  Mediocre  performance.  Meaningful 
correlation  with  other  features,  however. 
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Country  Reputation 


•  Country  =  spatial  grouping  over  editors 

•  Geo-location  data  maps  IP  ->  country 

•  Straightforward:  IP  resides  in  one  country 


good  rep.  — ->  poor  rep. 

CDF  of  Country  Reputation 


RANK 

COUNTRY 

%-OEs 

1 

Italy 

2.85% 

2 

France 

3.46% 

3 

Germany 

3.46% 

... 

... 

... 

12 

Canada 

11.35% 

13 

United  States 

11.63% 

14 

Australia 

12.08% 

OE-rate  (normalized)  for 
countries  with  100k+  edits 
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Classification  and  Performance 


CLASSIFICATION  & 
PERFORMANCE 
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MLTraining 


•  Calc,  features  for  all  edits. 
Normalize  onto  [0,1];  polarity 

•  SVM:  Support  Vector  Machine 

•  ISSUE:  {Unlabeled}  set  is  just 
that.  Very  low  cost  penalties 
so  no  over-compensation. 

•  Train  over  prior  subset  to 
classify  now  (100+  edits/sec). 


# 

FEATURE 

1 

Edit  time-of-day 

2 

Edit  day-of-week 

3 

Time-since  page  edited 

4 

Time-since  user  reg. 

5 

Time-since  last  user  OE 

6 

Rev.  comment  length 

7 

Article  reputation 

8 

Category  reputation 

9 

Editor  reputation 

10 

Country  reputation 

Review  of  features  used 
(only  IP-editors) 
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Performance 


•  ISSUE:  Edits  classified 
as  OE  but  in  {UnLbl} 
may  not  be  FPs: 

—  Manual  inspection 
—  Raw  vs.  adjusted 
—  Corpus  produced* 

•  Similar  performance 
to  NLP-efforts  [2] 

•  Use  as  an  intelligent 
routing  (IR)  tool 

•  Shown  steady-state 

•  http://www.cis.upenn.edu/~westand 
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Recall:  0  20  40  60  80  100 


Precision-recall  trade-off 


Recall:  %  OEs  classified  as  such 

Precision:  %  of  edits  classified  OE 
that  are  actually  vandalism 
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Conclusions 


•  Showed  spatio-temporal  properties  can  locate 
Wikipedia-vandalism  comparably  to  NLP 

-  Complementary;  still  some  advantages: 

•  Content/language  independent 

•  Harder  to  evade  (analysis  needed) 

•  Faster  (100+  edits/sec  vs.  5  edits/sec) 

•  Spatio-temporal  reputation  as  a  general-purpose 
technique  for  content-based  access  control? 

-  Email  spam:  SNARE  [1]  and  PreSTA  [5] 

—  This  work  shows  it  also  works  for  Wikipedia 
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STiki 


STiki  [4]:  A  real-time,  on-Wikipedia 
implementation  of  the  technique 
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STiki  Architecture 


EDIT  QUEUE:  Connection  between  server  and  client  side 

•  Populated:  Priority  insertion  based  on  vandalism  score 

•  Popped:  GUI  client  shows  likely  vandalism  first 

•  De-queued:  Edit  removed  if  another  made  to  same  page 
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Client  Demonstration 


STiki 

Client 

Demo 
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STiki  Performance 


•  Competition  inhibits  maximal  performance 

-  Metric:  Hit-rate  (%  of  edits  displayed  that  are  vandalism) 

—  Offline  analysis  shows  it  could  be  50%+ 

-  Competing  (often  autonomous)  tools  make  it  =10% 

•  STiki  successes  and  use-cases 

-  Has  reverted  over  3500+  instances  of  vandalism 

-  May  be  more  appropriate  in  less  patrolled  installations 

•  Any  of  Wikipedia's  foreign  language  editions 

•  Corporate  Wiki's  and  other  small  installations 

-  Embedded  vandalism:  That  escaping  initial  detection. 
Median  age  of  STiki  revert  is  4.25  hours,  200x  conventional 
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Alternative  Code  Uses 


•  All  code  is  available  [4]  and  open  source  (Java) 

•  Backend  (server-side)  re-use 

-  Large  portion  of  MediaWiki  API  implemented  (bots) 

-  Trivial  to  add  new  features  (including  NLP  ones) 

•  Frontend  (client-side)  re-use 

-  Useful  whenever  edits  require  human  inspection 

•  Data  re-use 

-  Corpus  building;  crowd-sourcing 

-  Incorporate  vandalism  score  into  more  robust  tools 
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Future  Direction:  Wiki-Spam 


•  Many  people  "see"  vandalism  and  do  nothing: 

-  Becomes  "embedded"  for  days/weeks  accumulating  views 

-  Traffic  spikes:  During  American  Idol  finale,  the  "Crystal 
Bowersox"  article  was  vandalized  for  just  28  seconds,  but 
12,000+  viewed  the  page  during  this  duration. 

-  Shows  evade-ability,  apathy,  or  both 

•  What  if  vandalism  was  spam? 

-  If  immature  vandalism  can  get  this  many  views,  what 
about  the  less  detectable  and  incentivized  spam? 

-  Could  it  be  more  profitable  than  email  spam? 

-  What  evasion  strategies  would  work  best? 
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